flowctl: re-tool `preview` on the runtime-next stack by jgraettinger · Pull Request #3117 · estuary/flow

jgraettinger · 2026-07-04T19:26:24Z

What & why

flowctl preview now runs on the runtime-next + shuffle stack instead of the
legacy V1 runtime::harness. It was developed on this branch as flowctl raw preview-next, then promoted to be the preview; the V1 preview harness — the
last consumer of runtime::harness — is deleted along with the now-dead
runtime/src/harness/* and its private exchange combinator.

The user-facing contract is unchanged: the CLI flags, stdout/stderr line shapes,
and exit codes that estuary/connectors CI depends on. --shards and
--debug-port are new, additive flags.

To get there, runtime-next's host-specific behavior — how a task publishes
documents, reads its sources, and emits logs — was factored into three factory
seams. Production installs journal-backed factories; preview installs
output-capturing / fixture-replay factories from flowctl. runtime-next is now
preview-agnostic, so the lone Task.preview flag — the "write documents to
output instead of collections" wire control — is removed from the runtime proto
(field 2 reserved); how a task publishes and logs is now a host concern of the
installed factories, not a wire flag. (max_transactions stays as the
transaction bound, re-documented as a general preview/harness control.)

The three host seams (`runtime-next`)

Each seam is a factory the leader and shard Services are generic over (static
RPITIT, not dyn):

publish — a PublisherFactory / JournalPublisher seam replaces the
Publisher enum (a Real variant plus a Preview variant that baked
NDJSON-to-stdout output into the crate); the publisher now carries only
document output, with its former observation calls moved to the logger seam.
shuffle (leader) — a new leader/shuffle.rs provides ShuffleSession(Factory)
- ShuffleServiceFactory, factoring out the leader's previously-inline
  shuffle-Session opening and checkpoint sourcing.
log — new Logger / LoggerFactory seam sinking both connector logs
(log) and structured LogEvents (event) — a #[non_exhaustive] enum
(persist / applied / inferred-schema / container lifecycle) carrying borrowed,
verbatim payloads so a host can intercept them structurally. LogEvent::to_log
flattens each event into the ops-log line it replaced, and event defaults to
that flattening, so a host overrides only the events it cares about. Production
shards install FnLoggerFactory; leaders and tests install
TracingLoggerFactory.

New harness helpers: seed_initial_connector_state / read_final_connector_state
seed and read shard zero's RocksDB by path; shuffle::log::BlockMeta is
re-exposed for segment writers.

Preview on the seams (`flowctl`)

publish: PreviewPublisherFactory writes captured/derived documents to
stdout as NDJSON.
log: PreviewLoggerFactory sinks the task log and powers --output-state
/ --output-apply (the legacy connectorState / applied lines) by
intercepting LogEvent::Persist / LogEvent::Applied structurally and
flattening everything else through LogEvent::to_log. The run-end final reduced
state comes from re-opening shard zero's RocksDB once the session loop releases
it.
shuffle: PreviewShuffleFactory selects per run between a live in-process
journal-reading Session and a --fixture replay (FixtureOpener).
--initial-state seeds shard zero's RocksDB via seed_initial_connector_state.
--delay, fixtures, and --sessions planning are unchanged.

Tests / soak

Deleted plans/runtime-v2/preview-harness.md, a closed bring-up doc for raw preview-next.
Soak README updated to the new command name.

Notes for reviewers

Depends on the incidental fixes in Incidental fixes: gazette Append retry, connector-init image, materialize tear-down #3121 only insofar as they were split off
this branch; the preview work does not require them, so the two can land in
either order.
Behavioral compatibility with legacy preview was verified before the cutover;
the promotion commit is mostly a module move (raw/preview_next/ → preview/)
plus deletion of the V1 harness.

…tories Factor runtime-next's host-specific behavior — how a task publishes, shuffles, and logs — out of the crate into three uniform, monomorphized host seams, each a factory the leader and shard `Service`s are generic over: - publish: replace the `Publisher` enum — a `Real` variant plus a `Preview` variant that baked NDJSON-to-stdout output into the crate — with a `PublisherFactory` / `JournalPublisher(Factory)` seam. The publisher now carries only document output; its former observation calls move to the Logger seam. - leader: factor the leader's inline shuffle-`Session` opening and checkpoint sourcing into a new `leader/shuffle.rs`, providing `ShuffleSession` / `ShuffleSessionFactory` (static RPITIT, not `dyn`) plus a `ShuffleServiceFactory`. - log: new `Logger` / `LoggerFactory` seam — the task's out-of-band log and event stream. A `Logger` sinks both the connector's own logs (`log`) and structured runtime `LogEvent`s (`event`): a `#[non_exhaustive]` enum (persist / applied / inferred-schema / container lifecycle) whose variants carry borrowed, verbatim payloads so a host may intercept them structurally. Its canonical `LogEvent::to_log` flattens each event into the ops-log line it replaced, and `event` defaults to that flattening — so a host overrides only to intercept, delegating the rest. Production shards install `FnLoggerFactory` (sinking to the task-log writer); leaders and tests install `TracingLoggerFactory`. Remove the now-unnecessary `Task.preview` flag from the runtime proto (reserving field 2): how a task publishes and logs is a host concern decided by the installed factories, not a protocol flag. `max_transactions` stays as the transaction bound, re-documented as a general preview/harness control rather than tied to the removed flag. Make the `shard::rocksdb` module public — exposing `RocksDB` (with its `open`, `scan`, and now-public `put_connector_state_base`) and re-exporting its `DecodeError` — so a host harness can seed shard zero's RocksDB by path and read back its final reduced state itself, and re-expose `shuffle::log::BlockMeta` for harness segment writers.

Drive `flowctl raw preview-next` through runtime-next's three factory seams by installing preview-specific implementations of each: - publish: an output-capturing `PreviewPublisherFactory` writes captured / derived documents to stdout as NDJSON. - log: a `PreviewLoggerFactory` sinks the task's log stream and powers `--output-state` / `--output-apply` (the legacy `connectorState` / `applied` lines) by intercepting runtime-next's `LogEvent::Persist` / `LogEvent::Applied` structurally — decoding state deltas from its tab-framed patch wire form — and flattening every other event through `LogEvent::to_log` into the same log handler as the connector's own logs. The `--output-state` run-end final reduced state is emitted by re-opening shard zero's RocksDB (`read_preview_state`) once the session loop has released it. - shuffle: a `PreviewShuffleFactory` enum selects per run between a live in-process journal-reading shuffle Session and a `--fixture` replay (`FixtureOpener`, a `ShuffleSessionFactory` that writes log segments and feeds synthetic checkpoint Frontiers). `--initial-state` seeds shard zero's RocksDB by path, and `--output-state`'s run-end read reopens it — both managed by preview itself through the now-public `runtime_next::shard::rocksdb` handle (the local `seed_preview_state` / `read_preview_state` helpers over `RocksDB::open`, `put_connector_state_base`, and `scan`). Each driver also encodes its task spec once and threads the shared bytes to its shards rather than re-encoding per shard. `--delay`, eager/streaming fixtures, and `--sessions` planning are unchanged.

`flowctl preview` now runs on the runtime-next + shuffle stack that was developed on this branch as `flowctl raw preview-next`, which the prior commits verified to be behaviorally compatible with the legacy preview. Move the `raw/preview_next/` module into `preview/`, delete the old V1-runtime `preview/{mod,journal_reader}.rs`, and drop the `raw preview-next` command wiring. The CLI flag surface, stdout/stderr line shapes, and exit-code contract that the estuary/connectors CI harnesses depend on are preserved; `--shards` and `--debug-port` are additive. The old preview was the sole consumer of `runtime::harness` (and its private `exchange` combinator), so remove that now-dead module too. Update the soak README to the new command name, and delete the runtime-v2 preview harness guide: a closed, point-in-time document describing `raw preview-next` as it stood during bring-up.

williamhbaker

A couple of questions

williamhbaker · 2026-07-06T20:01:05Z

+                schema,
+                ..
+            } => {
+                fields.push(("collection", json_field(collection_name)));


Wanting to confirm this field name...should it be collection_name? Since the L1 inferred schemas rollup is looking for that:

flow/ops-catalog/data-plane-template.flow.yaml

Line 60 in b98d27c

$fields->>'collection_name' as collection_name,

williamhbaker · 2026-07-06T20:04:15Z

+                // The patch payload is valid JSON (a tab-delimited JSON array;
+                // see `crate::patches`), so it embeds verbatim.
+                fields.push(("patches", persist.connector_patches_json.clone()));
+                (ops::LogLevel::Debug, "persisted connector-state delta")


Will these get filtered out for info and above level tasks, since they don't go through Tokio's tracing subscriber?

jgraettinger changed the title ~~Johnny/preview~~ flowctl: re-tool preview on the runtime-next stack Jul 5, 2026

jgraettinger force-pushed the johnny/preview branch 4 times, most recently from 036c2b9 to f6adb49 Compare July 6, 2026 15:26

jgraettinger added 3 commits July 6, 2026 12:33

jgraettinger force-pushed the johnny/preview branch from f6adb49 to 53cacd6 Compare July 6, 2026 16:34

jgraettinger mentioned this pull request Jul 6, 2026

Incidental fixes: gazette Append retry, connector-init image, materialize tear-down #3121

Merged

jgraettinger marked this pull request as ready for review July 6, 2026 18:21

jgraettinger requested review from dgreer-dev and williamhbaker July 6, 2026 18:21

williamhbaker reviewed Jul 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

flowctl: re-tool `preview` on the runtime-next stack#3117

flowctl: re-tool `preview` on the runtime-next stack#3117
jgraettinger wants to merge 3 commits into
masterfrom
johnny/preview

jgraettinger commented Jul 4, 2026 •

edited

Loading

Uh oh!

williamhbaker left a comment

Uh oh!

williamhbaker Jul 6, 2026

Uh oh!

williamhbaker Jul 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jgraettinger commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What & why

The three host seams (runtime-next)

Preview on the seams (flowctl)

Tests / soak

Notes for reviewers

Uh oh!

williamhbaker left a comment

Choose a reason for hiding this comment

Uh oh!

williamhbaker Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

williamhbaker Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jgraettinger commented Jul 4, 2026 •

edited

Loading

The three host seams (`runtime-next`)

Preview on the seams (`flowctl`)